{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactive model evaluation with Trident Chemwidgets\n",
    "\n",
    "In this tutorial we'll build on the [*Introduction to Graph Covolutions* tutorial](https://github.com/deepchem/deepchem/blob/master/examples/tutorials/Introduction_to_Graph_Convolutions.ipynb) to show how you can use the [Trident Chemwidgets](https://github.com/tridentbio/trident-chemwidgets) (TCW) package to interact with and test the model you've trained.\n",
    "\n",
    "Evaluating models on new data, including corner cases, is a critical step toward model deployment. However, generating new molecules to test in an interactive way is rarely straightforward. TCW provides several tools to help subset larger datasets and draw new molecules to test against your models. You can find the full documentation for the Trident Chemwidgets library [here](https://www.trident.bio/trident-chemwidgets/html/index.html). \n",
    "\n",
    "## Colab\n",
    "\n",
    "This tutorial and the rest in this sequence are designed to be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/Interactive_model_evaluation_with_Trident_Chemwidgets.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing the prerequisites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 211
    },
    "colab_type": "code",
    "id": "3Jv2cmnW91CF",
    "outputId": "bd523c54-3038-4654-89ad-356ad1e207ca",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --pre deepchem\n",
    "!pip install trident-chemwidgets seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial, you'll need Trident Chemwidgets version 0.2.0 or greater. We can check the installed version with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2.1\n"
     ]
    }
   ],
   "source": [
    "import trident_chemwidgets as tcw\n",
    "print(tcw.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Throughout this tutorial, we'll use the convention `tcw` to call the classes from the Trident Chemwidgets package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the data\n",
    "\n",
    "We'll start out by loading the Tox21 dataset and extracting the predefined train, validation, and test splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "id": "JMi2V8Jncj1W",
    "outputId": "56ab5eb6-07be-4d8f-c19b-88d1f73f2f46"
   },
   "outputs": [],
   "source": [
    "import deepchem as dc\n",
    "\n",
    "tasks, datasets, transformers = dc.molnet.load_tox21(featurizer='GraphConv')\n",
    "train_dataset, valid_dataset, test_dataset = datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use RDKit to calculate some additional features for each of the training examples. Specifically, we'll compute the [logP](https://en.wikipedia.org/wiki/Partition_coefficient) and [molecular weight](https://en.wikipedia.org/wiki/Molecular_mass) of each molecule and return this new data in a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[10:14:08] WARNING: not removing hydrogen atom without neighbors\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smiles</th>\n",
       "      <th>logp</th>\n",
       "      <th>mwt</th>\n",
       "      <th>split</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CC(O)(P(=O)(O)O)P(=O)(O)O</td>\n",
       "      <td>-0.9922</td>\n",
       "      <td>206.027</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CC(C)(C)OOC(C)(C)CCC(C)(C)OOC(C)(C)C</td>\n",
       "      <td>4.8172</td>\n",
       "      <td>290.444</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OC[C@H](O)[C@@H](O)[C@H](O)CO</td>\n",
       "      <td>-2.9463</td>\n",
       "      <td>152.146</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CCCCCCCC(=O)[O-].CCCCCCCC(=O)[O-].[Zn+2]</td>\n",
       "      <td>2.1911</td>\n",
       "      <td>351.802</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CC(C)COC(=O)C(C)C</td>\n",
       "      <td>1.8416</td>\n",
       "      <td>144.214</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     smiles    logp      mwt  split\n",
       "0                 CC(O)(P(=O)(O)O)P(=O)(O)O -0.9922  206.027  train\n",
       "1      CC(C)(C)OOC(C)(C)CCC(C)(C)OOC(C)(C)C  4.8172  290.444  train\n",
       "2             OC[C@H](O)[C@@H](O)[C@H](O)CO -2.9463  152.146  train\n",
       "3  CCCCCCCC(=O)[O-].CCCCCCCC(=O)[O-].[Zn+2]  2.1911  351.802  train\n",
       "4                         CC(C)COC(=O)C(C)C  1.8416  144.214  train"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import rdkit.Chem as Chem\n",
    "from rdkit.Chem.Crippen import MolLogP\n",
    "from rdkit.Chem.Descriptors import MolWt\n",
    "import pandas as pd\n",
    "import torch\n",
    "\n",
    "data = []\n",
    "\n",
    "for dataset, split in zip(datasets, ['train', 'valid', 'test']):\n",
    "    for smiles in dataset.ids:\n",
    "        mol = Chem.MolFromSmiles(smiles)\n",
    "        logp = MolLogP(mol)\n",
    "        mwt = MolWt(mol)\n",
    "        data.append({\n",
    "            'smiles': smiles,\n",
    "            'logp': logp,\n",
    "            'mwt': mwt,\n",
    "            'split': split\n",
    "        })\n",
    "        \n",
    "mol_data = pd.DataFrame(data)\n",
    "\n",
    "mol_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-dimensional distributions\n",
    "\n",
    "We can examine one-dimensional distributions using a histogram. Unlike histograms from static plotting libraries like Matplotlib or Seaborn, the TCW Histogram provides interactive functionality. TCW enables subsetting of the data, plotting chemical structures in a gallery next to the plot, and saving a reference to the subset portion of the dataframe. Unfortunately, this interactivity comes at the price of portability, so we have included screenshots for this tutorial in addition to providing the code to generate the interactive visuals. If you run this tutorial yourself (either locally or on Colab), you'll be able to display and interact with full demo plots.\n",
    "\n",
    "In the plot below, you can see the histogram of the molecular weight distribution from the combined dataset on the left. If you click and drag within plot area in the live widget, you can subset a portion of the distribution for further examination. The background of the selected portion will turn gray and the selected data points will be shown in teal within the bars of the plot. The x axis of the Histogram widget is compatible with either numeric or date data types, which makes it a convenient choice for splitting your ML datasets based on a property or the date the experimental data were collected.\n",
    "\n",
    "<img src=\"./assets/trident_chemwidgets_data/Histogram.png\" alt=\"Histogram example\" width=\"1000\"/>\n",
    "\n",
    "To generate an interactive example of the widget, run the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0698e9c966b481c95add09ae6deab94",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Histogram(data={'points': [{'smiles': 'CC(O)(P(=O)(O)O)P(=O)(O)O', 'x': 206.027, 'index': 0}, {'smiles': 'CC(C…"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist = tcw.Histogram(data=mol_data, smiles='smiles', x='mwt')\n",
    "hist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you select subset of the data by clicking and dragging, you can view the selected structures in the gallery to the right by pressing the `SHOW STRUCTURES` button beneath the plot. You can extract this subset of the original dataframe by pressing `SAVE SELECTION` and accessing the `hist.selection` property as shown in the next cell. This workflow is convenient for applications like data splitting based on a single dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smiles</th>\n",
       "      <th>logp</th>\n",
       "      <th>mwt</th>\n",
       "      <th>split</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [smiles, logp, mwt, split]\n",
       "Index: []"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist.selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two- or three-dimensional distributions\n",
    "\n",
    "In addition to histograms, TCW also has provides a class for scatter plots. The Scatter class is useful when comparing two or three dimensions or your data. As of v0.2.0, TCW Scatter supports the use of the x and y axes as well as the color of each point (`hue` keyword) to represent either continuous or discrete variables. Just like in the Histogram example, you can click and drag within the plot area to subset along the x and y axes. The Scatter widget also supports dates along the x, y, and hue axes.\n",
    "\n",
    "In the image below, we have selected a portion of dataset with large molecular weight values, but minimal training examples (displayed points in orange), to demonstrate how the Scatter widget can be useful for outlier identification. In addition to selection by bounding box, you can also hover over individual points to display a drawing of the underlying structure.\n",
    "\n",
    "<img src=\"./assets/trident_chemwidgets_data/Scatter.png\" alt=\"Scatter example\" width=\"1000\"/>\n",
    "\n",
    "To generate an interactive example of the widget, run the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f62315257b50449c99067a611ea81cfa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Scatter(data={'points': [{'smiles': 'CC(O)(P(=O)(O)O)P(=O)(O)O', 'x': 206.027, 'y': -0.9922000000000002, 'hue'…"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scatter = tcw.Scatter(data=mol_data, smiles='smiles', x='mwt', y='logp', hue='split')\n",
    "scatter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you select subset of the data by clicking and dragging, you can view the selected structures in the gallery to the right by pressing the `SHOW STRUCTURES` button beneath the plot. You can extract this subset of the original dataframe by pressing `SAVE SELECTION` and accessing the `scatter.selection` property as shown in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smiles</th>\n",
       "      <th>logp</th>\n",
       "      <th>mwt</th>\n",
       "      <th>split</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [smiles, logp, mwt, split]\n",
       "Index: []"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scatter.selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a GraphConvModel\n",
    "\n",
    "Now that we've had a look at the training data, we can train a GraphConvModel to predict the 12 Tox21 classes. We'll replicate the training procedure exactly from the [*Introduction to Graph Covolutions* tutorial](https://github.com/deepchem/deepchem/blob/master/examples/tutorials/Introduction_to_Graph_Convolutions.ipynb). We'll train for 50 epochs, just as in the original tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 245
    },
    "colab_type": "code",
    "id": "Y9n3jTNHcj1a",
    "outputId": "2caab2e5-5e5a-4f97-a440-753692341d35",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2595431746140894\n"
     ]
    }
   ],
   "source": [
    "# The next line filters pytorch warnings relevant to memory consumption.\n",
    "# To see these warnings, comment the next line.\n",
    "import warnings; warnings.filterwarnings('ignore')\n",
    "\n",
    "# Now we'll set the pytorch seed to make sure the results of this notebook are reproducible\n",
    "import torch\n",
    "torch.manual_seed(27)\n",
    "n_tasks = len(tasks)\n",
    "num_features = train_dataset.X[0].get_atom_features().shape[1]\n",
    "model = dc.models.torch_models.GraphConvModel(n_tasks, mode='classification',number_input_features=[num_features,64])\n",
    "model.fit(train_dataset, nb_epoch=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a trained model, we can check AUROC values for the training and test datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.96\n",
      "Test set score: 0.69\n"
     ]
    }
   ],
   "source": [
    "metric = dc.metrics.Metric(dc.metrics.roc_auc_score)\n",
    "print(f'Training set score: {model.evaluate(train_dataset, [metric], transformers)[\"roc_auc_score\"]:.2f}')\n",
    "print(f'Test set score: {model.evaluate(test_dataset, [metric], transformers)[\"roc_auc_score\"]:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as in the original tutorial, we see that the model performs reasonably well on the predefined train/test splits. Now we'll use this model to evaluate compounds that are outside the training distribution, just as we might in a real-world drug discovery scenario."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating the model on new data\n",
    "\n",
    "One of the challenging first steps toward deploying an ML model in production is evaluating it on new data. Here, new data refers to both data outside the initial train/val/test distributions and also data that may not be already processed for use with the model.\n",
    "\n",
    "We can use the JSME widget provided by TCW to quickly test our model again some molecules of interest. We'll start with a known therapeutic molecule: ibuprofen. We can see that ibuprofen is not included in any of the datasets that we have evaluated our model against so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ibuprofen structure in Tox21 dataset: False\n"
     ]
    }
   ],
   "source": [
    "print(f\"Ibuprofen structure in Tox21 dataset: {'CC(C)CC1=CC=C(C=C1)C(C)C(=O)O' in mol_data['smiles']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simulate a drug discovery application, let's say you're a chemist tasked with identifying potential new therapeutics derived from ibuprofen. Ideally, the molecules you test would have limited toxicity. You've just developed the model above to predict the tox outcomes from Tox21 data and now you want to use it to do some first-pass screening of your derivatives. The standard workflow for a task like this might include drawing the molecules in a program like ChemDraw, exporting to SMILES format, importing into the notebook, then prepping the data and running it through your model.\n",
    "\n",
    "With TCW, we can shortcut the first few steps of that workflow by using the [JSME](https://jsme-editor.github.io/help.html) widget to draw molecules and convert to SMILES directly in the notebook. We can even use the `base_smiles` argument to specify a base molecular structure, which is great for generating derivatives. Here we'll set the base_smiles value to `'CC(C)CC1=CC=C(C=C1)C(C)C(=O)O'`, the SMILES string for ibuprofen. Below is a screenshot using JSME to generate a few derivative molecules to test against our toxicity model.\n",
    "\n",
    "<img src=\"./assets/trident_chemwidgets_data/JSME.png\" alt=\"JSME example\" width=\"1000\"/>\n",
    "\n",
    "To generate your own set of derivatives, run the cell below. To add a SMILES string to the saved set, click the `ADD TO SMILES LIST` button below the interface. If you want to regenerate the original base molecule, in this case ibuprofen, click the `RESET TO BASE SMILES` button below the interface. By using this button, it's easy to generate distinct derivatives from a shared starting structure.  Go ahead and create some ibuprofen derivatives to test against the tox model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d835df550dc4a15be371ef878bd9a83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "JSME(base_smiles='CC(C)CC1=CC=C(C=C1)C(C)C(=O)O')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jsme = tcw.JSME(base_smiles='CC(C)CC1=CC=C(C=C1)C(C)C(=O)O')\n",
    "jsme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the smiles using the `jsme.smiles` property. This call will return a list of the SMILES strings that have been added to the SMILES list of the widget (the ones shown in the molecule gallery to the right of the JSME interface)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(jsme.smiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure the rest of this notebook runs correctly, the following cell sets the new test SMILES set to the ones from the screenshot above in the case that you have not defined your own set using the widget. Otherwise, it will use the molecules you have drawn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell will provide a preset list of SMILES strings in case you did not create your own.\n",
    "if len(jsme.smiles) > 1:\n",
    "    drawn_smiles = jsme.smiles\n",
    "else:\n",
    "    drawn_smiles = [\n",
    "        'CC(C)Cc1ccc(C(C)C(=O)O)cc1',\n",
    "        'CC(C)C(S)c1ccc(C(C)C(=O)O)cc1',\n",
    "        'CCSC(c1ccc(C(C)C(=O)O)cc1)C(C)CC',\n",
    "        'CCSC(c1ccc(C(C)C(=O)O)cc1)C(C)C(=O)O',\n",
    "        'CC(C(=O)O)c1ccc(C(S)C(C)C(=O)O)cc1'\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we have to create a dataset that is compatible with our model to test these new molecules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "featurizer = dc.feat.ConvMolFeaturizer()\n",
    "loader = dc.data.InMemoryLoader(tasks=list(train_dataset.tasks), featurizer=featurizer)\n",
    "dataset = loader.create_dataset(drawn_smiles, shard_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can generate our predictions of positive results here and plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAGiCAYAAAB6c8WBAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKB9JREFUeJzt3X9UVXW+//HXAeXgb00S8CeaFZqKBcmgmf3gympcNN67psicJCxn5WCjsvQqlaLTJFaT45QmV0ttupk2reo2aXodSpuWNChGk+WPTI2uCWomGObBztnfP/rGdDaonDycfejzfLT2H3zOPvvzxvTw5v3+fPZ2WZZlCQAAGCvC6QAAAICzSAYAADAcyQAAAIYjGQAAwHAkAwAAGI5kAAAAw5EMAABgOJIBAAAMRzIAAIDhSAYAADAcyQAAAGHinXfeUWZmprp37y6Xy6XXXnvtgu/ZsmWLrrnmGrndbvXv31+rV68OeF6SAQAAwkRtba2SkpK0dOnSJp1/8OBBjRkzRjfeeKPKy8s1bdo03Xvvvdq0aVNA87p4UBEAAOHH5XLp1Vdf1dixY895zqxZs7R+/Xrt2rWrfuyOO+7QyZMntXHjxibPRWUAAIBm5PF4VFNT43d4PJ6gXLukpETp6el+YxkZGSopKQnoOq2CEk0QXBN/ndMh6J9fHnQ6BOV1v97pEFStb50OQbk643QIyjj5qdMh6OXogU6HoKmuKqdD0AdfHnA6BHVwt3U6BP3fwzc5HYIufSCw8nNzqT19qFmvf/Z48P7OFS75s+bPn+83VlBQoHnz5l30tSsrKxUbG+s3Fhsbq5qaGn3zzTdq06ZNk64TNskAAABhw+cN2qXy8/OVl5fnN+Z2u4N2/WAgGQAAoBm53e5m++EfFxenqir/6l1VVZU6duzY5KqARDIAAEBDls/pCJokLS1NGzZs8BvbvHmz0tLSAroOCwgBALDz+YJ3BODrr79WeXm5ysvLJX23dbC8vFwVFRWSvms5TJgwof78++67TwcOHNB//ud/as+ePXr66af10ksvafr06QHNS2UAAAAby6HKwI4dO3TjjTfWf/39WoPs7GytXr1aR44cqU8MJKlv375av369pk+frj/96U/q2bOnnnnmGWVkZAQ0L8kAAABh4oYbbtD5bv/T2N0Fb7jhBr3//vsXNS/JAAAAdgGW91s6kgEAAOxayALCYGEBIQAAhqMyAACAXRBvOtQSkAwAAGBHmwAAAJiEygAAAHbsJgAAwGxO3XTIKbQJAAAwHJUBAADsaBMAAGA4w9oEJAMAANgZdp8B1gwAAGA4KgMAANjRJgAAwHCGLSCkTQAAgOGoDAAAYEebAAAAw9EmAAAAJqEyAACAjWWZdZ8BkgEAAOwMWzNAmwAAAMNRGQAAwM6wBYQkAwAA2BnWJiAZAADAjgcVAQAAk1AZAADAjjYBAACGYwHh+R0/flwrV65USUmJKisrJUlxcXEaPny47r77bl166aVBDxIAADSfgJKB7du3KyMjQ23btlV6erquuOIKSVJVVZWefPJJLVy4UJs2bVJKSsp5r+PxeOTxePzGfJZPES6WMAAAwgBtgnO7//77ddttt6moqEgul8vvNcuydN999+n+++9XSUnJea9TWFio+fPn+43Fteul+A69AwkHAIDmYVibIKBfxT/44ANNnz69QSIgSS6XS9OnT1d5efkFr5Ofn6/q6mq/I7Z9z0BCAQAAQRJQZSAuLk6lpaVKTExs9PXS0lLFxsZe8Dput1tut9tvjBYBACBsGFYZCCgZmDFjhn7961+rrKxMN998c/0P/qqqKhUXF2vFihX6wx/+0CyBAgAQKjy18Dxyc3MVExOjP/7xj3r66afl9X73hxUZGank5GStXr1at99+e7MECgAAmkfAWwuzsrKUlZWls2fP6vjx45KkmJgYtW7dOujBAQDgCNoETdO6dWvFx8cHMxYAAMIDWwsBADCcYZUBlvADAGA4KgMAANjRJgAAwHC0CQAAgEmoDAAAYEebAAAAw9EmAAAAJqEyAACAnWGVAZIBAADsDFszQJsAAADDURkAAMCONgEAAIYzrE1AMgAAgJ1hlQHWDAAAYDgqAwAA2NEmAADAcLQJAACASagMAABgZ1hlgGQAAAA7y3I6gpCiTQAAgOGoDAAAYEebAAAAwxmWDNAmAADAcFQGAACw46ZDAAAYjjYBAACGs6zgHQFaunSpEhISFB0drdTUVJWWlp73/MWLF+vKK69UmzZt1KtXL02fPl1nzpwJaE6SAQAAwsS6deuUl5engoIC7dy5U0lJScrIyNDRo0cbPX/NmjWaPXu2CgoKtHv3bj377LNat26dHnjggYDmJRkAAMDO5wveEYBFixZp0qRJysnJ0cCBA1VUVKS2bdtq5cqVjZ6/bds2jRgxQnfeeacSEhI0evRojRs37oLVBDuSAQAA7IKYDHg8HtXU1PgdHo+nwZR1dXUqKytTenp6/VhERITS09NVUlLSaJjDhw9XWVlZ/Q//AwcOaMOGDfr5z38e0LcbNgsIP/rqM6dDCAvPn/zA6RD05Tc1ToegnK7DnA5BYzoPdDoE3fBF4x8AoRTVqrXTISgcbgxb4zntdAg6W7bf6RDk+fas0yG0OIWFhZo/f77fWEFBgebNm+c3dvz4cXm9XsXGxvqNx8bGas+ePY1e+84779Tx48d13XXXybIsffvtt7rvvvtoEwAAcNEsX9CO/Px8VVdX+x35+flBCXPLli1asGCBnn76ae3cuVOvvPKK1q9fr4cffjig64RNZQAAgHBh+YJXj3K73XK73Rc8LyYmRpGRkaqqqvIbr6qqUlxcXKPvmTNnju666y7de++9kqTBgwertrZWv/71r/Xggw8qIqJpv/NTGQAAIAxERUUpOTlZxcXF9WM+n0/FxcVKS0tr9D2nT59u8AM/MjJSkmQFsK2RygAAAHYO3XQoLy9P2dnZSklJ0bBhw7R48WLV1tYqJydHkjRhwgT16NFDhYWFkqTMzEwtWrRIV199tVJTU7V//37NmTNHmZmZ9UlBU5AMAABg59DtiLOysnTs2DHNnTtXlZWVGjp0qDZu3Fi/qLCiosKvEvDQQw/J5XLpoYce0uHDh3XppZcqMzNTjzzySEDzuqxA6gjNyB3dy+kQ5A2D209e2raT0yGExW6Cd8JgN8EzUc530Z5jN4EkVrB/78usRKdDUNd1ja9qD7Vv6w436/VPL7s/aNdqO/mpoF2ruVAZAADALogLCFsCkgEAAOzCoFIcSiQDAADYGZYMON8UBQAAjqIyAACAXXisrQ8ZkgEAAOxoEwAAAJNQGQAAwI6thQAAGM6hOxA6hTYBAACGozIAAIAdbQIAAMxmsZsAAACYhMoAAAB2tAkAADCcYbsJSAYAALAzrDLAmgEAAAxHZQAAADvDdhOQDAAAYEebAAAAmITKAAAAduwmAADAcLQJAACASagMAABgY9qzCUgGAACwo00AAABMQmUAAAA7wyoDJAMAANixtRAAAMMZVhkI+pqBzz//XBMnTjzvOR6PRzU1NX6HZZn1Bw8AQLgIejJw4sQJPffcc+c9p7CwUJ06dfI7vN6aYIcCAMCPYvmsoB0tQcBtgtdff/28rx84cOCC18jPz1deXp7fWMylAwMNBQCA5tFCfogHS8DJwNixY+Vyuc5b1ne5XOe9htvtltvtDug9AACgeQTcJoiPj9crr7win8/X6LFz587miBMAgNDx+YJ3tAABJwPJyckqKys75+sXqhoAABD2fFbwjhYg4DbBzJkzVVtbe87X+/fvr7fffvuiggIAAKETcDIwcuTI877erl07jRo16kcHBACA41rIb/TBwk2HAACwMa3dzYOKAAAwHJUBAADsaBMAAGA4kgEAAMzWUm4jHCysGQAAwHBUBgAAsDOsMkAyAACAXcu4i3DQ0CYAAMBwVAYAALAxbQEhyQAAAHaGJQO0CQAAMByVAQAA7AxbQEgyAACAjWlrBmgTAABgOCoDAADY0SYAAMBsprUJSAYAALAzrDLAmgEAAAxHZQAAABvLsMoAyQAAAHaGJQO0CQAAMByVAQAAbGgTAABgOsOSAdoEAAAYjsoAAAA2prUJqAwAAGBj+YJ3BGrp0qVKSEhQdHS0UlNTVVpaet7zT548qdzcXMXHx8vtduuKK67Qhg0bApqTygAAADZOVQbWrVunvLw8FRUVKTU1VYsXL1ZGRob27t2rbt26NTi/rq5O//Zv/6Zu3brp5ZdfVo8ePfTZZ5+pc+fOAc1LMgAAQJhYtGiRJk2apJycHElSUVGR1q9fr5UrV2r27NkNzl+5cqVOnDihbdu2qXXr1pKkhISEgOelTQAAgJ3lCtrh8XhUU1Pjd3g8ngZT1tXVqaysTOnp6fVjERERSk9PV0lJSaNhvv7660pLS1Nubq5iY2M1aNAgLViwQF6vN6BvN2wqA5Zl1hOizuVU3TdOh6AIl/M54u9bNfyHEmq91c7pEJQVn+p0CCr75v+cDkGfVn/hdAj6bfxIp0NQ1K9GOB2C7vn7VqdDCIlgtgkKCws1f/58v7GCggLNmzfPb+z48ePyer2KjY31G4+NjdWePXsavfaBAwf01ltvafz48dqwYYP279+v3/zmNzp79qwKCgqaHGPYJAMAAPwU5efnKy8vz2/M7XYH5do+n0/dunXT8uXLFRkZqeTkZB0+fFiPP/44yQAAABfD8rmCdi23292kH/4xMTGKjIxUVVWV33hVVZXi4uIafU98fLxat26tyMjI+rEBAwaosrJSdXV1ioqKalKMzteDAQAIM05sLYyKilJycrKKi4vrx3w+n4qLi5WWltboe0aMGKH9+/fL5/vXRPv27VN8fHyTEwGJZAAAgLCRl5enFStW6LnnntPu3bs1efJk1dbW1u8umDBhgvLz8+vPnzx5sk6cOKGpU6dq3759Wr9+vRYsWKDc3NyA5qVNAACAjWUFr00QiKysLB07dkxz585VZWWlhg4dqo0bN9YvKqyoqFBExL9+j+/Vq5c2bdqk6dOna8iQIerRo4emTp2qWbNmBTQvyQAAADZO3o54ypQpmjJlSqOvbdmypcFYWlqa3nvvvYuakzYBAACGozIAAIBNMHcTtAQkAwAA2Jh2HzySAQAAbEyrDLBmAAAAw1EZAADAxrTKAMkAAAA2pq0ZoE0AAIDhqAwAAGBDmwAAAMM5dTtip9AmAADAcFQGAACwcfLZBE4gGQAAwMZHmwAAAJiEygAAADamLSAkGQAAwIathQAAGI47EAIAAKNQGQAAwIY2AQAAhmNrIQAAMAqVAQAAbNhaCACA4dhNAAAAjEJlAAAAG9MWEJIMAABgY9qaAdoEAAAYjsoAAAA2pi0gdCQZ8Hg88ng8fmOWZcnlMqssAwAIT6atGQi4TfDNN9/o3Xff1ccff9zgtTNnzujPf/7zBa9RWFioTp06+R0+76lAQwEAoFlYlitoR0sQUDKwb98+DRgwQNdff70GDx6sUaNG6ciRI/WvV1dXKycn54LXyc/PV3V1td8REdkh8OgBAMBFCygZmDVrlgYNGqSjR49q79696tChg0aMGKGKioqAJnW73erYsaPfQYsAABAufJYraEdLEFAysG3bNhUWFiomJkb9+/fXX//6V2VkZGjkyJE6cOBAc8UIAEBIWUE8WoKAkoFvvvlGrVr9a82hy+XSsmXLlJmZqVGjRmnfvn1BDxAAADSvgHYTJCYmaseOHRowYIDf+JIlSyRJt956a/AiAwDAIS2lvB8sAVUG/v3f/10vvvhio68tWbJE48aNk2Xa5kwAwE8OuwnOIz8/Xxs2bDjn608//bR8Pt9FBwUAAEKHOxACAGBj2q+1JAMAANhYahnl/WDhQUUAABiOygAAADY+w9bCkwwAAGDjM6xNQDIAAIANawYAAIBRqAwAAGDD1kIAAAxHmwAAABiFygAAADa0CQAAMJxpyQBtAgAADEdlAAAAG9MWEJIMAABg4zMrF6BNAACA6agMAABgw7MJAAAwnGEPLSQZAADAjq2FAADAKFQGAACw8blYMwAAgNFMWzNAmwAAAMNRGQAAwMa0BYQkAwAA2HAHQgAAYBSSAQAAbHxyBe0I1NKlS5WQkKDo6GilpqaqtLS0Se9bu3atXC6Xxo4dG/CcJAMAANhYQTwCsW7dOuXl5amgoEA7d+5UUlKSMjIydPTo0fO+79ChQ5oxY4ZGjhwZ4IzfIRkAAKAZeTwe1dTU+B0ej6fRcxctWqRJkyYpJydHAwcOVFFRkdq2bauVK1ee8/per1fjx4/X/Pnz1a9fvx8VY9gsILyqSx+nQ9CHJw45HYK6te3sdAiqqDl/BhoKK3qfdjoEfXXE6Qikn1XudjoE1Xm/dTqEsPDCyQ+cDkH/vPcrp0PQwTPHnA4hJIK5gLCwsFDz58/3GysoKNC8efP8xurq6lRWVqb8/Pz6sYiICKWnp6ukpOSc1//d736nbt266Z577tHf//73HxVj2CQDAACEi2BuLczPz1deXp7fmNvtbnDe8ePH5fV6FRsb6zceGxurPXv2NHrtd999V88++6zKy8svKkaSAQAAbIJ5B0K3293oD/+LderUKd11111asWKFYmJiLupaJAMAAISBmJgYRUZGqqqqym+8qqpKcXFxDc7/9NNPdejQIWVmZtaP+Xzf1TRatWqlvXv36rLLLmvS3CwgBADAxucK3tFUUVFRSk5OVnFx8b/i8PlUXFystLS0BucnJibqww8/VHl5ef1x66236sYbb1R5ebl69erV5LmpDAAAYOPU7Yjz8vKUnZ2tlJQUDRs2TIsXL1Ztba1ycnIkSRMmTFCPHj1UWFio6OhoDRo0yO/9nTt3lqQG4xdCMgAAQJjIysrSsWPHNHfuXFVWVmro0KHauHFj/aLCiooKRUQEv6hPMgAAgI2TDyqaMmWKpkyZ0uhrW7ZsOe97V69e/aPmJBkAAMDG4kFFAADAJFQGAACwcbJN4ASSAQAAbExLBmgTAABgOCoDAADYBPN2xC0ByQAAADbBfGphS0AyAACADWsGAACAUagMAABgY1plgGQAAAAb0xYQ0iYAAMBwVAYAALBhNwEAAIYzbc0AbQIAAAxHZQAAABvTFhCSDAAAYOMzLB2gTQAAgOGoDAAAYGPaAkKSAQAAbMxqEpAMAADQgGmVAdYMAABgOCoDAADYcAdCAAAMx9ZCAABgFCoDAADYmFUXIBkAAKABdhMAAACjOFIZ8Hg88ng8fmM+y6cIF7kJAMB5LCC8gN27d2vVqlXas2ePJGnPnj2aPHmyJk6cqLfeeqtJ1ygsLFSnTp38jqra/ws0FAAAmoUVxKMlCCgZ2Lhxo4YOHaoZM2bo6quv1saNG3X99ddr//79+uyzzzR69OgmJQT5+fmqrq72O2Lb9fzR3wQAAPjxAkoGfve732nmzJn68ssvtWrVKt15552aNGmSNm/erOLiYs2cOVMLFy684HXcbrc6duzod9AiAACEC18Qj5YgoJ/AH330ke6++25J0u23365Tp07pl7/8Zf3r48eP1z//+c+gBggAQKj5ZAXtaAkCXkDocn13j8aIiAhFR0erU6dO9a916NBB1dXVwYsOAAAHtIwf4cETUGUgISFBn3zySf3XJSUl6t27d/3XFRUVio+PD150AACg2QVUGZg8ebK8Xm/914MGDfJ7/c0339RNN90UnMgAAHBIS+n1B0tAycB999133tcXLFhwUcEAABAOLMMaBSzhBwDAcDybAAAAG9oEAAAYrqVsCQwW2gQAABiOygAAADZm1QVIBgAAaIA2AQAAMAqVAQAAbNhNAACA4Uy76RDJAAAANqZVBlgzAACA4agMAABgQ5sAAADD0SYAAABGoTIAAICNz6JNAACA0cxKBWgTAABgPCoDAADYmPZsApIBAABsTNtaSJsAAADDURkAAMDGtPsMkAwAAGDDmgEAAAzHmgEAAGAUKgMAANiwZgAAAMNZht2OmDYBAABhZOnSpUpISFB0dLRSU1NVWlp6znNXrFihkSNHqkuXLurSpYvS09PPe/65kAwAAGDjkxW0IxDr1q1TXl6eCgoKtHPnTiUlJSkjI0NHjx5t9PwtW7Zo3Lhxevvtt1VSUqJevXpp9OjROnz4cEDzkgwAAGDjC+Lh8XhUU1Pjd3g8nkbnXbRokSZNmqScnBwNHDhQRUVFatu2rVauXNno+S+88IJ+85vfaOjQoUpMTNQzzzwjn8+n4uLigL7fsFkz8NFXnzkdQlg4UnvC6RDCwnOf93A6BK3xfOp0CDp9tvEPjFBqHen8x4TP53z/9stvapwOQXXtv3U6BB2srnQ6hBansLBQ8+fP9xsrKCjQvHnz/Mbq6upUVlam/Pz8+rGIiAilp6erpKSkSXOdPn1aZ8+e1SWXXBJQjM7/KwcAIMwE8z4D+fn5ysvL8xtzu90Nzjt+/Li8Xq9iY2P9xmNjY7Vnz54mzTVr1ix1795d6enpAcVIMgAAgE0w70Dodrsb/eEfbAsXLtTatWu1ZcsWRUdHB/RekgEAAMJATEyMIiMjVVVV5TdeVVWluLi48773D3/4gxYuXKi//e1vGjJkSMBzs4AQAAAby7KCdjRVVFSUkpOT/Rb/fb8YMC0t7Zzve+yxx/Twww9r48aNSklJ+VHfL5UBAABsnLoDYV5enrKzs5WSkqJhw4Zp8eLFqq2tVU5OjiRpwoQJ6tGjhwoLCyVJjz76qObOnas1a9YoISFBlZXfLfBs37692rdv3+R5SQYAALBx6kFFWVlZOnbsmObOnavKykoNHTpUGzdurF9UWFFRoYiIfxX1ly1bprq6Ov3yl7/0u05juxXOh2QAAIAwMmXKFE2ZMqXR17Zs2eL39aFDh4IyJ8kAAAA2wdxN0BKQDAAAYMODigAAgFGoDAAAYEObAAAAwzm1m8AptAkAADAclQEAAGx8hi0gJBkAAMDGrFSANgEAAMajMgAAgA27CQAAMBzJAAAAhuMOhAAAwChUBgAAsKFNAACA4bgDIQAAMAqVAQAAbExbQEgyAACAjWlrBmgTAABgOCoDAADY0CYAAMBwtAkAAIBRqAwAAGBj2n0GSAYAALDxsWYAAACzmVYZYM0AAACGozIAAIANbYIfwbIsuVyuYFwKAADHmdYmCEoy4Ha79cEHH2jAgAFNOt/j8cjj8fiNkVAAAOCMgJKBvLy8Rse9Xq8WLlyorl27SpIWLVp03usUFhZq/vz5fmMRER0U2apjIOEAANAsaBOcx+LFi5WUlKTOnTv7jVuWpd27d6tdu3ZN+u0+Pz+/QWLRNaZpVQUAAJobbYLzWLBggZYvX64nnnhCN910U/1469attXr1ag0cOLBJ13G73XK73X5jtAgAAHBGQFsLZ8+erXXr1mny5MmaMWOGzp4921xxAQDgGJ9lBe1oCQK+z8C1116rsrIyHTt2TCkpKdq1axe/1QMAflKsIP7XEvyo3QTt27fXc889p7Vr1yo9PV1erzfYcQEAgBC5qK2Fd9xxh6677jqVlZWpT58+wYoJAABHWZbP6RBC6qLvM9CzZ0/17NkzGLEAABAWfC2kvB8s3I4YAAAbq4Us/AsWHlQEAIDhqAwAAGBDmwAAAMPRJgAAAEahMgAAgE1LuXNgsJAMAABg01LuHBgstAkAADAclQEAAGxMW0BIMgAAgI1pWwtpEwAAYDgqAwAA2NAmAADAcGwtBADAcKZVBlgzAACA4agMAABgY9puApIBAABsaBMAAACjUBkAAMCG3QQAABiOBxUBAACjUBkAAMCGNgEAAIZjNwEAADAKlQEAAGxMW0BIMgAAgA1tAgAADGdZVtCOQC1dulQJCQmKjo5WamqqSktLz3v+X/7yFyUmJio6OlqDBw/Whg0bAp6TZAAAgDCxbt065eXlqaCgQDt37lRSUpIyMjJ09OjRRs/ftm2bxo0bp3vuuUfvv/++xo4dq7Fjx2rXrl0BzeuywqQWEuXu6XQIYbGVpHWk852bs95vnQ5Bv4+/0ekQtMbzqdMh6OMTFU6HwN/J/y/C5XI6BA2/NNHpEPTu0d1OhyBJ+rbucLNev1VUj6Bdq/bUAXk8Hr8xt9stt9vd4NzU1FRde+21WrJkiSTJ5/OpV69euv/++zV79uwG52dlZam2tlZvvPFG/djPfvYzDR06VEVFRU0P0voJOHPmjFVQUGCdOXOGGIghbOIgBmIghvCMIdQKCgosSX5HQUFBg/M8Ho8VGRlpvfrqq37jEyZMsG699dZGr92rVy/rj3/8o9/Y3LlzrSFDhgQU408iGaiurrYkWdXV1cRADGETBzEQAzGEZwyhdubMGau6utrvaCwZOnz4sCXJ2rZtm9/4zJkzrWHDhjV67datW1tr1qzxG1u6dKnVrVu3gGJ0vv4HAMBP2LlaAuGEBYQAAISBmJgYRUZGqqqqym+8qqpKcXFxjb4nLi4uoPPPhWQAAIAwEBUVpeTkZBUXF9eP+Xw+FRcXKy0trdH3pKWl+Z0vSZs3bz7n+efyk2gTuN1uFRQUOFqGIYbwiSFc4iAGYiCG8IwhnOXl5Sk7O1spKSkaNmyYFi9erNraWuXk5EiSJkyYoB49eqiwsFCSNHXqVI0aNUpPPPGExowZo7Vr12rHjh1avnx5QPOGzdZCAAAgLVmyRI8//rgqKys1dOhQPfnkk0pNTZUk3XDDDUpISNDq1avrz//LX/6ihx56SIcOHdLll1+uxx57TD//+c8DmpNkAAAAw7FmAAAAw5EMAABgOJIBAAAMRzIAAIDhfhLJQKCPewymd955R5mZmerevbtcLpdee+21kM39vcLCQl177bXq0KGDunXrprFjx2rv3r0hjWHZsmUaMmSIOnbsqI4dOyotLU1vvvlmSGOwW7hwoVwul6ZNmxayOefNmyeXy+V3JCaG/uEyhw8f1q9+9St17dpVbdq00eDBg7Vjx46QzZ+QkNDgz8Hlcik3NzdkMXi9Xs2ZM0d9+/ZVmzZtdNlll+nhhx8O+XPqT506pWnTpqlPnz5q06aNhg8fru3btzfrnBf6XLIsS3PnzlV8fLzatGmj9PR0ffLJJyGN4ZVXXtHo0aPVtWtXuVwulZeXB3V+BKbFJwOBPu4x2Gpra5WUlKSlS5eGZL7GbN26Vbm5uXrvvfe0efNmnT17VqNHj1ZtbW3IYujZs6cWLlyosrIy7dixQzfddJN+8Ytf6KOPPgpZDD+0fft2/dd//ZeGDBkS8rmvuuoqHTlypP549913Qzr/V199pREjRqh169Z688039fHHH+uJJ55Qly5dQhbD9u3b/f4MNm/eLEm67bbbQhbDo48+qmXLlmnJkiXavXu3Hn30UT322GN66qmnQhaDJN17773avHmznn/+eX344YcaPXq00tPTdfhw8z1170KfS4899piefPJJFRUV6R//+IfatWunjIwMnTlzJmQx1NbW6rrrrtOjjz4atDlxEQJ6kkEYGjZsmJWbm1v/tdfrtbp3724VFhaGPBZJDZ425YSjR49akqytW7c6GkeXLl2sZ555JuTznjp1yrr88sutzZs3W6NGjbKmTp0asrkLCgqspKSkkM3XmFmzZlnXXXedozHYTZ061brsssssn88XsjnHjBljTZw40W/sP/7jP6zx48eHLIbTp09bkZGR1htvvOE3fs0111gPPvhgSGKwfy75fD4rLi7Oevzxx+vHTp48abndbuvFF18MSQw/dPDgQUuS9f777zfL3GiaFl0ZqKurU1lZmdLT0+vHIiIilJ6erpKSEgcjc1Z1dbUk6ZJLLnFkfq/Xq7Vr16q2tjbgW2IGQ25ursaMGeP39yKUPvnkE3Xv3l39+vXT+PHjVVFREdL5X3/9daWkpOi2225Tt27ddPXVV2vFihUhjeGH6urq9N///d+aOHGiXC5XyOYdPny4iouLtW/fPknSBx98oHfffVe33HJLyGL49ttv5fV6FR0d7Tfepk2bkFeMvnfw4EFVVlb6/fvo1KmTUlNTjf7cNF2Lvh3x8ePH5fV6FRsb6zceGxurPXv2OBSVs3w+n6ZNm6YRI0Zo0KBBIZ37ww8/VFpams6cOaP27dvr1Vdf1cCBA0Maw9q1a7Vz585m78meS2pqqlavXq0rr7xSR44c0fz58zVy5Ejt2rVLHTp0CEkMBw4c0LJly5SXl6cHHnhA27dv129/+1tFRUUpOzs7JDH80GuvvaaTJ0/q7rvvDum8s2fPVk1NjRITExUZGSmv16tHHnlE48ePD1kMHTp0UFpamh5++GENGDBAsbGxevHFF1VSUqL+/fuHLI4fqqyslKRGPze/fw3madHJABrKzc3Vrl27HPmt48orr1R5ebmqq6v18ssvKzs7W1u3bg1ZQvD5559r6tSp2rx5c4PfxELlh791DhkyRKmpqerTp49eeukl3XPPPSGJwefzKSUlRQsWLJAkXX311dq1a5eKioocSQaeffZZ3XLLLerevXtI533ppZf0wgsvaM2aNbrqqqtUXl6uadOmqXv37iH9c3j++ec1ceJE9ejRQ5GRkbrmmms0btw4lZWVhSwG4EJadJvgxzzu8adsypQpeuONN/T222+rZ8+eIZ8/KipK/fv3V3JysgoLC5WUlKQ//elPIZu/rKxMR48e1TXXXKNWrVqpVatW2rp1q5588km1atVKXq83ZLF8r3Pnzrriiiu0f//+kM0ZHx/fIAEbMGBAyNsVkvTZZ5/pb3/7m+69996Qzz1z5kzNnj1bd9xxhwYPHqy77rpL06dPr3/AS6hcdtll2rp1q77++mt9/vnnKi0t1dmzZ9WvX7+QxvG97z8b+dzED7XoZODHPO7xp8iyLE2ZMkWvvvqq3nrrLfXt29fpkCR99//C4/GEbL6bb75ZH374ocrLy+uPlJQUjR8/XuXl5YqMjAxZLN/7+uuv9emnnyo+Pj5kc44YMaLB1tJ9+/apT58+IYvhe6tWrVK3bt00ZsyYkM99+vRpRUT4f8RFRkbK5/OFPBZJateuneLj4/XVV19p06ZN+sUvfuFIHH379lVcXJzf52ZNTY3+8Y9/GPW5CX8tvk1wocc9Nrevv/7a77e+gwcPqry8XJdccol69+4dkhhyc3O1Zs0a/c///I86dOhQ3/fr1KmT2rRpE5IY8vPzdcstt6h37946deqU1qxZoy1btmjTpk0hmV/6rj9rXyfRrl07de3aNWTrJ2bMmKHMzEz16dNHX3zxhQoKChQZGalx48aFZH5Jmj59uoYPH64FCxbo9ttvV2lpqZYvXx7wI00vls/n06pVq5Sdna1WrUL/UZOZmalHHnlEvXv31lVXXaX3339fixYt0sSJE0Max6ZNm2RZlq688krt379fM2fOVGJiYrN+Rl3oc2natGn6/e9/r8svv1x9+/bVnDlz1L17d40dOzZkMZw4cUIVFRX64osvJKk+gY2Li6NC4QSntzMEw1NPPWX17t3bioqKsoYNG2a99957IZv77bfftiQ1OLKzs0MWQ2PzS7JWrVoVshgmTpxo9enTx4qKirIuvfRS6+abb7b+93//N2Tzn0uotxZmZWVZ8fHxVlRUlNWjRw8rKyvL2r9/f8jm/95f//pXa9CgQZbb7bYSExOt5cuXhzyGTZs2WZKsvXv3hnxuy7Ksmpoaa+rUqVbv3r2t6Ohoq1+/ftaDDz5oeTyekMaxbt06q1+/flZUVJQVFxdn5ebmWidPnmzWOS/0ueTz+aw5c+ZYsbGxltvttm6++eag/3+6UAyrVq1q9PWCgoKgxoGm4RHGAAAYrkWvGQAAABePZAAAAMORDAAAYDiSAQAADEcyAACA4UgGAAAwHMkAAACGIxkAAMBwJAMAABiOZAAAAMORDAAAYLj/B9suswni2bpcAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions = model.predict(dataset, transformers)[:, :, 1]\n",
    "\n",
    "import seaborn as sns\n",
    "sns.heatmap(predictions, vmin=0, vmax=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can get the predicted most toxic compound/assay result for further inspection. Below we extract the highest predicted positive hit (most toxic) and display the assay name, SMILES string, and an image of the structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most toxic result (predicted): SR-MMP, CCSC(c1ccc(C(C)C(=O)O)cc1)C(C)CC\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcIAAACWCAIAAADCEh9HAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3deVwT19oH8CckQARlMwuLbKW4oGDrAmq1VkRoUerW4i62KNaNtm5V6VVsVdQreCuoFGuLaFV6X21pUVRUbF1ZBCQoWkkhoEgSQEAEQkLm/WO8qUXBhWQmE5/vxz8yA5nzi/p55mTmzDksgiAAIYTQyzKiOwBCCDEbllGEEOoULKMIIdQpWEYRQqhTsIwihFCnYBlFCD0DjufpGJZRhFBHmpubfXx8vvnmG7qD6C8sowihjqxcuTI7OzshIUGhUNCdRU+xsLuOEGrP6dOn/f392Wz2pUuXBg8eTHccPYW9UYTQ09XW1oaGhhIEsX79eqyhHcDeKELo6aZPn37o0KGhQ4eeP3+ezWbTHUd/YRlFCD3F0aNHJ0+ebG5unpub27NnT7rj6DX8Uo8QaquioiIsLAwAoqOjsYY+E5ZRhNA/EAQxd+7c6upqf39/spiijmEZRQj9w65du9LS0ng8XmJiIovFojsOA+C1UYTQ38Ri8RtvvNHQ0JCcnBwcHEx3HGbA3ijz1NbW0h0BGSaVSjVjxoyGhoZZs2ZhDX1+WEaZpKioaNCgQR988AHdQZBh2rRpU2ZmpoODAz76+ULwSz2T1NfXOzs719bWXrp0aejQoXTHQQYlNzd3yJAhra2t6enpvr6+dMdhEuyNMomFhcWCBQsA4N///jfdWZBBaW5unj17tlKp/PTTT7GGvijsjTKMTCZzcXFpbm4WiUR9+/alOw4yEOHh4bGxsX369Ll69WqXLl3ojsMw7MjISLozoBdgbm5+7969rKyshw8fTpgwge44L6axsXHz5s0bN268e/dudXV1TU1NU1MTm83mcrl0R3ulnTlzZsmSJRwOJzU11dnZme44zIO9UeYpLy93c3MDgNu3bzPrP72Xl5dIJHrqj6ytre3s7Kytra2tre3t7TWvNZsCgYDD4VAc+FVQV1fn5eVVVla2YcOGiIgIuuMwEpZRRpo9e/b+/fuXLFmyY8cOurM8r1OnTgUEBABA3759PTw8mpubq6qqqqqqpFJpfX39M99uZGTE+x8+ny8UCjWbQqGQz+fzeLzu3bubmJjo/qMYlBkzZhw8eBDnH+kMLKOMVFRU1K9fPxMTk5KSEltbW7rjPFttba2Xl1d5efmKFSu2bt3a5qctLS1yubyqqkomk8nlcvK1XC6XyWRkqSX3PE9DxsbGa9eu/fLLL3XwIQwQOf+ImZlZXl4ePjv/0rCMMtWECRNSUlIiIiI2bNhAd5ZnmzZt2uHDh4cNG/bHH3+8dJenqanp3r17FRUV9+/fv3///pOvKysryf/PhYWFDL3/plRWqlTVpqbORkZddd1WRUWFl5dXdXX17t27P/nkE103Z8CwjDJVVlaWj4+PpaWlRCKxtLSkO05HDh06NH36dHNz87y8PHd3d3JnfHy8qanp41/Su3Xr1vm2nJycysvLly5dGh0d3fmjUUmpvCcWT25qKuBweCqV3Mlpd/fus3XXHEEQ48aNO378uL+//4kTJ/DZ+c7AMspgvr6+GRkZW7ZsWblyJd1Z2lVRUeHp6VlTU5OQkDBv3jzN/m7dujU0NLT55fZuNGleC4XCjjuz2dnZ3t7eFhYWEonEyspKJx9JNySSUIXir9dfP2ZkZNbaep8gWjkcnu6a27Vr16JFi6ytrQsKCnr06KG7hl4FWEYZLD093d/fXygUlpSU6OdYP4Igxo4dm5aWFhAQkJaWpunyEATxxRdfkFc8yUuiMpnsyar6JDabrenACgQC8s4Sj8cbMGDAsGHDyN/x8/M7c+ZMVFTUqlWrdPjZtO3WrRFduvRzctpNQVs4/4h2YRlltqFDh165cmXXrl3k0036JjY2Njw8nMfjiUSiZ94KUygU1dXV9//nqZdBZTJZa2vrk+/95JNPdu9+VIDOnDnj5+cnEAhKS0v18+zyVBUV6yorN9varuTxQk1MXAAAgKirO2FsLOBwBBwOz8hIO59FpVKNGDHiypUrM2fO3L9/v1aO+YrDMsps5J1WV1fXP//8U9+GVd68eXPgwIGNjY0//fTThx9+qJVjqlQqzY17qVSquY8/YsSIqVOnan5t2LBhly9f3rlz58KFC7XSLiUIuXy3XJ7Q1CSytBzr6ppEEOpr17prfmxkZM7h8IyNbTkcHvnH2FjI4fDJ1zIZn8cTPM/15a+++mrdunUODg4ikcja2lqXn+hVgWWU2QiC8PT0vH79+oEDB2bMmEF3nL+pVKrhw4dnZmaGhIQkJiZS3PrPP/88adIkJyen4uJiY2NjilvvpMbG/JKSqWZmg3r02FZa+pFKJVWp5CpVlVrd3N5bWCyOt3draythampqY2PT5smFx1+fPXv2s88+Iwji+PHj7777LpWfy4BhGWW8xMTEjz76qE+fPoWFhUZG+jLXTGRk5Pr163v06FFQUEB9l0dzdtm/f//MmTMpbr3zKiu31NQc8vDIf3ynWt2gVMpUKplKVaVSValUcqVSSr5uaWkeO1Ysl8sfPnz4PMf39vbOzMzUTfZXEZZRxlMqle7u7hKJ5Ndffw0KCqI7DsBjU66dPn161KhRtGTYt2/fnDlz9O3s0oG6ulQzszeNjR1Uquri4rFcbm8Xl8QXPUhTU9OTTy5orn5UVVVJJJI333wzNTXVxsZGBx/iFYVl1BDs2LHj008/9fHxuXLlCt1ZoLm5edCgQdevX6d38Kbm7JKSkvL+++/TFeP5lZZ+fP9+MovFUasfdus2xtU1icPh0x0KPR8CMd/Dhw/5fD4AnDt3ju4sxKJFiwDAw8OjsbGR3iTkhAPe3t70xuiYWt388GG2Zqul5W5rK81/b+hFMeDLDnomMzOz8PBwAIiKiqI3yenTp3ft2mVsbLxv3z7aBxvNmzfP1tY2KysrIyOD3iQduHv3XzdvDpXLdwEAAMvY2F5bA5sQZbCMGoglS5ZYWlqePHny6tWrdGWora39+OOPCYJYv379oEGD6IqhweVylyxZAnpwdmlPQ8MFmSwGAMzMBtKdBb08LKMGwtLSMiwsDAC2bNlCV4YFCxaUl5cPHTpUfx5OXbx4sZWVVXp6ek5ODt1Z2lKrG0pL5xBEq53dl+bmPnTHQS8Py6jhWLZsWZcuXY4cOXLjxg3qWz969Ojhw4fNzc337dunP9NWWlhYzJ8/HwA2b95Md5a2ysqWKBRiM7MBtrZr6M6COgXLqOEQCoUhISFqtZr6++MVFRXktCMxMTGaOZz0xOeff96lS5eff/6ZlrNLe2prU6qrE42MuC4uSSwWwx4QQG1gGTUoK1eu5HA4+/fvLysro6xRgiDmzp1bU1MTEBDw+BxOekIoFM6ZM0etVm/bto3uLI+oVPKysvkA4ODw7y5dGDkvKnocllGD4urqGhwcrFQqt2/fTlmjO3fuTEtL4/F4iYmJ+jlt5YoVKzgczoEDByQSCd1ZAAAkkk+USqmFhZ9AsIjuLEgLsIwamoiICCMjo4SEBLlcTkFzYrF49erVALBr1y69Xc7E1dV1ypQpFJ9d2lNV9V1t7VE228rZ+XsAfTzroBeFZdTQeHh4BAYGNjY2xsXF6botlUo1Y8aMhoaG2bNna2sOJx0hzy579uyh5uzSHoWi5M6dpQDg5LTbxMSRxiRIi3CdegPk4uLy/fffZ2dnSySS7OxskUgkFosrKysbGhrUarWpqam27qR//fXXP/74Y48ePVJSUmgfbN8xPp+fm5tbWFhoamrq6+tLSwa1ulUsnqhQ3La2nmJvH0lLBqQL+Ey9ASouLu7Xr5+VlZVUKn3qL3C53A4W6iBfP3NaJs38I+np6XQVphdC++pVUVFRp07937p16uHDz3A4ODOI4cAyamjUavU777xz/vx5X1/fyZMna6b2IRfqIF8rlcpnHsfMzOzxJeB5PJ5AIBAIBORrCwuL4ODgGzduMGvxuFGjRp07d27r1q0rVqyguOn8/HwfHx+lUpmamhoYGEhx60insIwamg0bNvzrX/9ycHAoKChobzK0pqamNqt0tNmsqKiora3tuKEuXbrY29sXFhZyuVwdfA6dOHXqVEBAAPWrVykUCm9v74KCgkWLFlFwzRpRDMuoQcnLyxsyZIhSqTx27Nh7773XmUM9fPiwzVSVUqlUswhdSUmJTCZ76623zp8/T/5+fHz8jh07zp49q7f360mDBw/OycmJj48nn26ixvLly6Ojo93c3PLz87t21fkC9IhqdE4vhbSqubnZ09MTAMLDw3XdVm1tLXl5MScnh9wzefJkAPjiiy903XQn/fe//wWA1157TalUUtPi+fPn2Ww2h8O5cuUKNS0iimEZNRyfffYZAPTu3ZuaiT6XL18OAMHBweRmXl4ei8WysLC4f/8+Ba2/tNbWVg8PDwD48ccfKWjuwYMHbm5uALB27VoKmkO0wDJqIP744w8jIyMOh5OZmUlNi/fu3eNyuWw2+9atW+SeMWPGAMDGjRupCfDS9u7dCwAeHh6tra26bmvOnDkAMGDAgJaWFl23heiC10YNQV1dXf/+/SUSyfr169euXUtZu/Pnz09ISJg7d+6ePXsAICMjw9fXt3v37hKJxNzcnLIYzyQWi2NjYzdu3EimamlpcXFxuXfv3uDBg93d3cmxB3w+XygU8v6HXE2gk1JSUiZMmMDlcrOzs/v169f5AyL9hGXUEMyePXv//v0DBw68fPkylesJ//XXX7169WKxWMXFxU5OTgDw1ltvXbp0KTY2dvHixZTF6Fhra+vbb7996dKlzz//PCYmBgAaGhp69uypVCqrqqo6eCOXy21vXC256ejo2MHftlwu9/T0lEql33zzDbk2ATJUWEYZ75dffpk4caKZmVlubm6vXr0obn369OmHDh3SVCiy/+Xo6FhcXGxiYkJxmKfatGlTRESEvb29SCQiR4CFhYXt2bOnb9++W7Zsqa2t1SylqRmHIJfLq6urn+fg3bt3f+q4Wh6Pt3379pMnT44ePTo9PV0/Z2xB2oJllNlkMpmnp6dMJtu5c+fChQupD1BQUPDGG2+YmZmVlpbyeDyCILy8vAoLCxMTE0NCQqjP08aTI8BOnjz53nvvmZiYZGVleXl5dfBecvzsU8fVkq87fpCBz+c3NjYWFRU5OuKz8wYOyyizjRs37tixY2PGjDl58iRdXR4yw7p168j5GQ4cODBr1qzevXtfv36d3gXiFQrF4MGDRSLRkiVLyFVCq6qqPD09Kysro6Ojly5d2snjEwRR9ZjHx9WWl5dfvHjR1NS0rKyMx+Np49MgPUbj7S3USfHx8QBgZWVVVlZGY4zLly8DgI2NTX19PUEQKpWKHOJz9OhRGlMRBPH5558DQO/evR8+fEjuIaehGj58uEql0nXr48aNAxzn9GrAMspUYrG4W7duAHDo0CG6sxAjRowAgOjoaHJz586dADB48GAaI2lGgGkGvScmJgKAhYVFaWkpBQGuXLny+NkFGTAso4xE3n0GgGnTptGdhSAI4vjx4wBga2vb1NREEERTU5OdnR0AnD59mpY8tbW1zs7OABAZGUnuKS8vJ+esSkxMpCwG+W+0bds2ylpEtMAyykgbNmwAAAcHh+rqarqzPDJgwAAA2LNnD7lJLg0/evRoWsLMnj0bHhv03traOmrUKAAYP348lTHS0tIeP7sgQ4VllHlyc3NNTExYLNbx48fpzvK3w4cPA4Cbmxt52bGurs7KygoALl26RHGSX375BQC4XG5hYSG5h1zMjs/nV1ZWUhxm4MCBAJCQkEBxu4hKWEYZRjP/yJIlS+jO8g8qlYpcWjk5OZncQ67RNHHiRCpjSKVSoVAIALGxseSe69evk3Pi/fbbb1QmISUnJwO1M6Eg6mEZZZgn7z7rj4SEBADo37+/Wq0mCEIqlZqZmbFYLJFIRFkG8v64n58fmaGlpWXQoEEAMG/ePMoyPE6lUvXs2VNP7gQiHcEy2j6JhGgzZKeoiEhL+3tTqSROnCD+8x8iPp4oKKAg0ZN3n/VKS0sL+Uio5moD+Ujo7NmzqQnw7bffthkBRvaIXV1dabxdTk444OXlRVZ2ZHiwjLbv8GHCzu4fe2JiiIEDH70uLyc8PQlnZ2LGDGL8eMLcnPjoI0KXoxGfvPush8gFRUaMGEFulpWVmZiYGBsbl5SU6Lrpv/76q80IsEuXLrHZbCMjo3Pnzum69Q5ozi7Hjh2jMQbSHSyj7eu4jPr5EaNHEw0NjzZv3CCsrIhvvtFdnFmzZgHAwIED9XnKtYaGBvKhnfPnz5N7yJvmixcv1mm7mhFgkyZN0iQhr9WuWrVKp00/D3LCgSFDhtAdBOkEltH2dVBGS0sJAKLNN+tVqwgvLx1l+fnnnwHAzMzs5s2bOmpCW9atWwcAY8eOJTeLioqMjIy4XO69e/d01+imTZsAwN7eXjMCLCwsDADeeOMNhUKhu3af05NnF2RIsIy27/Bhols3YsOGv/8EBj4qo8eOESwW0WaS+Z9+IjgcQq0mjhwhUlKIixeJW7cIbUwFL5VKBQIBAMTFxXX+aLpWXV1NLjeUm5tL7pkwYQIArFmzRkct5uXltRkBduLECRaLZWpqeu3aNR01+qLICQcCAwPpDoK0D6cmaV9yMoSGwuMLn+XmwoMHkJMDKSnwwQegUMDjU28cOwZBQdDSAs7OUFHx935jY+DzgccDPh8EgkevebwKR0expaVmXrUOJhYJCAg4deqUo6OjRCJhxJRrS5cu3b59+7Rp0w4ePAgA2dnZ3t7eFhYWEomEHEyqRZr5RxYvXhwbGwuPzT+ybdu2ZcuWabe5l1ZTU+Pi4vLgwYOrV6+Sjyogw0F3HddjHXypz84mAIg2E4LExRH29gRBEPPnE0FBxNChhLs7YWFBADz1z+WRIzX/CkZGRgKBwMPDY+TIkZMnT164cOHatWt37Nhx8OBBPz8/8nfYbHZxcTFVH75T7ty5Y2Jiwmaz//zzT3KPr68vAERFRWm9LXKiJjc3twcPHpB7qJx/5IWQUadOnUp3EKRlWEbb10EZVSoJOzvi66///pFaTQweTMyd+5TjNDcTd+4Q+fnEyZPEgQPEf/5DfPklMX/+mfDw4cOH9+7du+OJ1MjuZ+/evQFg4cKFuvzA2hQaGgoA8+fPJzfT09MBQCAQaHe5vScX3dy3bx8AdO3aVQ9POZrVqzRnF2QYsIy2r+M79YcPE6amxPr1xNWrxNmzxMSJhFBIlJe/XFMqlaqyslIkEmVkZCQnJ8fFxb3zzjsAYGdn5+7u/uWXX968eZO8UVNRUdG5T0WR4uJiNpttamp69+5dck9ISMjBgwe12EOsq6sjR4CtW7eO3KOZf+SHH37QVivaNXfuXAAICwujOwjSJiyj7Ttzhpgw4R97kpP/0d88fZoYN4547TWid28iLIyQSLTYeHV1NTkK8urVq+SeSZMmgX4M33lO5JfrFStW6Oj45Oz6j88/Ql46oHj+kRfy5NkFGQAso/qLvD0yZcoUcjM3N5cRC8Fr5Ofns1gsc3PzqqoqXRw/Pj6ex+MVFRWRm+TIf1rmH3khwcHBALB8+XK6gyCtwTKqv568lEbebtq0aRO9wZ4fGVgzhlTrNBML3Lhxg5x/5MiRIzpqS1s0Zxe5XE53FqQdbHI4G9JDXbt2LSsry8nJaW5uDgoKAoAePXokJSWRg3uoXEj5pdXV1aWlpd2+fTsmJiY1NfW33347ffp0ZmZmQUFBcXHxvXv36uvrVSoV+cDoSxyffJdKpQoKCpJIJHPnzv3iiy+0/SG0zNbWNjMz88aNG127dh352GgNxFw4blSvkQvBs9lssVjs4OAAAMOGDbt8+XJcXNyiRYvoTvdc/P39ydv0HeNyue0tB6/ZtLOze+qw2eXLl0dHR7u6ul67do28oKznzp8///bbb9vY2EgkEvJRBcRoWEb13dSpU5OTk5ctW0bOPUyuSu/o6CgWixnRIQUAsVgslUrVarVmRXjyBbmaJrmUZnNz8zOPw+VyeTwen88XCoXkMwsEQRQUFGRkZLBYrIyMDAZ17kaMGHHhwoXt27d/9tlndGdBnYVlVN9du3btzTfffHIh+KSkJHKyEsPQ1NTU3nLwmpXiyXtHT327j48PuYQcUxw7dmzcuHE9evQQi8UmJiZ0x0GdgmWUAQIDA9PS0iIjI8lZP5KSkkJCQvr06VNYWEjvQvAUa2pqqqqqkslkZH+2qqoqNzc3Kyvr/fffX7duHeO+HQ8YMCAvL2/v3r0ff/wx3VlQp2AZZYA2l9KUSmXPnj3f7dZtU1SU9dixdKdDL+nQoUPTp093c3O7desWm82mOw56ea9QX4a5RowYMXz48JqaGnIedWNj4/yIiN0ikfVXX9EdDb284OBgd3d3sVhMzoKImAvLKDOsXr3ahM22PnMGWloAwHLmTLCzg6wsOHuW7mjoJbHZbPIJi6+//hq/FDIafqlnBoIgmt591+zUKfjuOwgNBQCIioI1a8DPD55jOBHSTwqFwtbWVq1W19fXkyO6njrSi3zN5/OZMjbjVYNllDkOH4Zp08DNDW7dAjYb6uvB2Rlqa+HyZRgyhO5w6GWcO3du9OjRXC63sbHxeX6fz+drJqgVCoWaTT6fLxAIyE287089LKPM0doKffrA7dvw00/w4YcAAKtXw+bNMGkSHDlCdzj0wurq6vr37y+RSCIjI9esWUOOPZDL5ZqxtOS4WnKALUmtVj/zsBYWFuS4WrKqCgSCd999l0EjapkIyyijJCTA/PnQvz/k5QGLBTIZuLhAczOIRNC3L93h0IsJCQlJSkoaMGDAlStXnvPbelNTU5uxtG2G2d65c6elpaXNuzZv3qz/z8gyGpZRRlEo4LXXoKICTpyAgAAAgEWLYNcumDMHfviB7nDoBaSkpEyYMIHL5ebk5PTV6imwtrZWJpNp+rBSqXT06NE+Pj5abAK1gWWUabZtgxUrYORIOHcOAKCsDF5/HQDg9m1wdqY1GXpeMpnMy8tLKpXGxsYuXryY7jios3DAE9MsWADdu8Pvv8PFiwAATk4wZQoolRATQ3cy9LxCQ0OlUqmfnx9T5pdBHcMyyjTm5rBwIQDAli2P9qxZA0ZG8N13IJPRmAs9p4SEhNTUVCsrq++//54R67yiZ8IyykCffgpdu0JqKuTlAQD06QNBQdDYCDt20J0MPUNJScny5csBYPfu3Y6OjnTHQdqBZZSBuneH0FAgCNi27dGe1asBABITQaWiMRfqmFqtnjNnzoMHDyZNmjR16lS64yCtwVtMzHTnDri5QWsr3Lz56BbT3r0wfjx0uFYzoldUVNSaNWvs7e1FIpGNjQ3dcZDWYG+UmXr0gJkzobUV4uMf7QkNxRqqz/Lz8yMjI1ks1nfffYc11MBgb5Sx/vwTMjJgzhwwNYWcHEhMhJISsLQEPz+YNQvw4Wt9olAovL29CwoKFi1aFBcXR3ccpGVYRplv/34IDYV582DYMKishB074PXX4cQJrKT6Y9myZTExMW5ubvn5+YybXho9E5ZRhqutBUdHiIoCzShuqRR69YKNGwHHJOqHCxcuvPPOOywW68KFC/g0kUHCa6MMl54OKhWEhf29RyiEadMAZwLWD/X19TNnzmxtbY2IiMAaaqiwjDKcRAKOjtBmbjR3dygtpScP+qfw8HCJRDJgwICIiAi6syBdwTLKcGw2KJVtdyqVgGv76IGUlJR9+/ZxudykpCSccdmAYRlluJ494e5dePDgHztv3IBevWgKhB6Ry+Xz588HgK1bt2p3Diekb7CMMtzo0WBjA1u3/r2HnNd5xgz6MiGA/80/Mnr0aJzDyeBx6A6AOofLhaQkmDABrl+Ht96CykrYuxeCgyE4mO5kr7Q9e/b89ttvOP/IKwIHPBkEiQQOHYKSEujaFQICwN+f7kCvtJKSkv79+z948ODgwYPTpk2jOw7SOSyjCGmTWq329fX9/fffJ02adATXyHo14LVRhLRp69atv//+u729fUJCAt1ZEEWwN4qQ1ly/fn3QoEEKhSI1NTUwMJDuOIgi2BtFSDvq6+vHjx/f3Ny8YMECrKGvFOyNIqQdw4cPv3jxIo/HKy0tNTc3pzsOog72RhHSjvLycgCYPHky1tBXDZZRhLTD3t4eAPLz8+kOgqiGX+oR0g6xWOzp6dnc3CwSifDpz1cK9kYR0g43N7ePPvqIIIhtmqUG0asBe6MIaU1ZWdnrr78OALdv33Z2dqY7DqII9kYR0honJ6cpU6YolcqYmBi6syDqYG8UIW0qKirq168fl8stKSkRCAR0x0FUwN4oQtrUp0+foKCgxsbGHTt20J0FUQR7owhpWVZWlo+Pj6WlpUQisbS0pDsO0jnsjSKkZd7e3qNGjaqrq/v222/pzoKogL1RhLQvPT3d399fKBSWlJR06dKF7jhIt7A3ipD2jRkzZtCgQVKpdN++fXRnQTqHZRQhnVi1ahUAbN26VaVS0Z0F6RaWUYR0YuLEiR4eHiUlJcnJyXRnQbqFZRQhnTAyMlq+fDkAbNy4Ua1W0x0H6RDeYkJIV5RKpbu7u0Qi+fXXX4OCguiOg3QFe6MI6YqxsfHSpUsBYOPGjXRnQTqEvVGEdKixsdHFxUUul587d27kyJF0x0E6gb1RhHTIzMwsPDwcAKKioujOgnQFe6MI6VZdXZ2zs3NdXV1OTs7AgQPpjoO0D3ujCOmWpaVlWFgYAGzZsoXuLEgnsDeKkM5JpVJXV1eFQiESiTw8POiOg7QMe6MI6ZxQKAwJCVGr1dHR0XRnQdqHvVGEqFBSUtKzZ08Wi1VcXOzk5ER3HKRN2BtFiAqurq7BwcFKpXL79u10Z0Fahr1RhChy48YNT09PLpdbWlrK5/PpjoO0BnujCFHEw8MjMDCwsbExLi6O7ixIm7A3ihB1MjMzhwwZYmNjU1pa2q1bN7rjIO3A3ihC1PHx8Xn77bdrampwfRFDwqE7AEKvloiICFZN2wQAAAA8SURBVAcHh4CAALqDIK3BL/UIIdQp+KUeIYQ6BcsoQgh1CpZRhBDqFCyjCCHUKVhGEUKoU7CMIoRQp/w/nHGIceLT+XMAAAFxelRYdHJka2l0UEtMIHJka2l0IDIwMjQuMDMuNgAAeJx7v2/tPQYgEABiRgYIEIbiBkY2hgSQODOEZmISYFAA0RAuMyObgwaIZmFzyADRIAE4AyoDUwk3CCLOAaGZOBjAGhgZsRiBoRXiBhifG+heRiYGJmagcQwsrBlMrGwJbOwZTOwcDBycDBxcDFzcGkxcPArsvAm8fBlMzPwM/AIM/IIMgkIMfCwJTiDPsrHw8bKzsYq/Q/I8g3Aw1zqHuytUHECctzsnOSzLX20PYr866urwR2Y3mC2nyu9gfj3TDsTefsfW/uOCzWD2euOnezd8TtwPYsusP7ufbaYTmL35Cf+B860SYL1LJuUfqHxgDmY/8e85cHk/C9iu7ZKTD2x4zAxW76VWe8Cp8T2YveHDoQNcQif3gdgX2jft/6PyCqxX1PGs7bLyW2C2s5GFQ3PIBLB6qQ/dDjcZF4DZFRMZHXykxQ+A2HPnaTkwPs4Fs8UANzteG895NhgAAAHgelRYdE1PTCByZGtpdCAyMDI0LjAzLjYAAHicfZRLjtswDIb3OYUuUIFvkctJMiiKYhKgk/YO3ff+KCkjtQcjVDYFS/5C8/Erp1bjx/X77z/t36Dr6dQa/OeOiPaLAeD01uqhnV+/fru1y+Pl/Ny53H/eHu8No26o6yP78ri/PXewXZp2YR8GjboOchgNOsyx/5KSk66uGNawMweLLjhu7407BBBocmLoRAtO0h91YjPBfE3knAF85jQ56EPQJHPprOC8is+S+wIdEQO1ntzJaBXhKBK7gQVYkZk5ysqnF5lBijNLvtfQEatkokDOKJnnJ0cmbbEAEYqUzi5DKQsAQwh8ReJGaph7VJiKWXteodTu9flQFaPKzQcirwJFLtSyP+iO5ZWDBtoKla1OHDJGLrtTkMqK3HqEAJStxj7Axli6tNl0Hw7m5RtpZEVX5Jhy48GgUSQpgC59+vQJJApS3WJDmJL6RMYkzTQYZ7vQwleae71dP5yR7dSc77frfmrqov1w5KLxfgYwTXapY5ruis5Fs124kjZ2dUqa7xLEtNiFhmVHPc0NPMiGakI6qAPnDh9EIOVYDq2WmlAPLa2Y0Q6twzmNQ4u2HT+0AucUh5KX1z33quyxjrV+/nPl8+kv6xPxZUy6UJIAAAD/elRYdFNNSUxFUyByZGtpdCAyMDI0LjAzLjYAAHicHZC7jQMxDERbcbgLyAL/HywuUgEOrgS14eJNKiMfyOEM1/pf18a997Wuda/r73N/7r3xPu16fS+dwuE2aKpT4HhkaigNnMzJMh6ekNC9GAaNhyaxmQyYRME2HpguaJJFWCHYx/OGiYiJOqqKICMtiNPAEpu5AUoPlpoEc+1qqmfpv7n0+CCvg5aFZHKIa5kEF8JDNC2ytRQVk89iqoqNOhSO2MjKPEb0GCc52rHBKe6VKCjbWJkFIOqMDubRGcMDvJWQXKO/ws5wtEmhipoBkgpIkw1BGphpGekAaBnj/v4A2sNN+4ptAQoAAAAASUVORK5CYII=",
      "text/plain": [
       "<rdkit.Chem.rdchem.Mol at 0x7fd1a1d36880>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "mol_idx, assay_idx = np.unravel_index(predictions.argmax(), predictions.shape)\n",
    "smiles = drawn_smiles[mol_idx]\n",
    "\n",
    "print(f'Most toxic result (predicted): {train_dataset.tasks[assay_idx]}, {smiles}')\n",
    "mol = Chem.MolFromSmiles(smiles)\n",
    "mol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interpreting the model's predictions\n",
    "\n",
    "Often predictions alone are insufficient to decide whether to move forward with costly experiments. We might also want some metric or metrics that allow us to interpret the model's output.\n",
    "\n",
    "Building on the tutorial [Calculating Atomic Contributions for Molecules Based on a Graph Convolutional QSAR Model](https://github.com/deepchem/deepchem/blob/master/examples/tutorials/Atomic_Contributions_for_Molecules.ipynb), we can calculate the relative contribution of each atom in a molecule to the predicted output value. This attribution strategy enables us to determine whether the molecular features that a chemist may identify as important and those most affecting the predictions are in alignment. If the chemist's interpretation and the model's interpretation metrics are consistent, that may indicate that the model is a good fit for the task at hand. However, the inverse is not necessarily true either. A model may have the capacity to make accurate predictions that a trained chemist cannot fully understand. This is just one tool in a machine learning practitioner's toolbox.\n",
    "\n",
    "We'll start by using the built-in `per_atom_fragmentation` argument for the `ConvMolFeaturizer`. This will generate a list of ConvMol objects that have each had a single atom removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "featurizer = dc.feat.ConvMolFeaturizer(per_atom_fragmentation=True)\n",
    "mol_list = featurizer(smiles)\n",
    "loader = dc.data.InMemoryLoader(tasks=list(train_dataset.tasks),\n",
    "                                featurizer=dc.feat.DummyFeaturizer())\n",
    "dataset = loader.create_dataset(mol_list[0], shard_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then run these predictions through the model and retrieve the predicted values for the molecule and assay specified in the last section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_molecule_prediction = predictions[mol_idx, assay_idx]\n",
    "fragment_predictions = model.predict(dataset, transformers)[:, assay_idx, 0]\n",
    "contributions = pd.DataFrame({\n",
    "    'Change in predicted toxicity': \n",
    "    (full_molecule_prediction - fragment_predictions).round(3)\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the InteractiveMolecule widget from TCW to superimpose the contribution scores on the molecule itself, allowing us to easily asses the relative importance of each atom to the final prediction. If you click on one of the atoms, you can retrieve the contribution data in a card shown to the right of the structure. In this panel you can also select a variable by which to color the atoms in the plot.\n",
    "\n",
    "<img src=\"./assets/trident_chemwidgets_data/InteractiveMolecule.png\" alt=\"InteractiveMolecule example\" width=\"1000\"/>\n",
    "\n",
    "You can generate the interactive widget by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf0f74ebc81841c18351f0ed108a934e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "InteractiveMolecule(data=[{'Change in predicted toxicity': 0.847000002861023}, {'Change in predicted toxicity'…"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tcw.InteractiveMolecule(smiles, data = contributions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j1FrVn88cj17"
   },
   "source": [
    "# Wrapping up\n",
    "\n",
    "In this tutorial, we learned how to incorporate Trident Chemwidgets into your DeepChem-based ML workflow. While TCW was built with molecular ML workflows in mind, the library also works well for general cheminformatics notebooks as well. \n",
    "\n",
    "## Star Trident Chemwidgets on [GitHub](https://github.com/tridentbio/trident-chemwidgets)\n",
    "If you find the Trident Chemwidgets package helpful please give it a ⭐️ on GitHub. Starring the project helps it grow and find new audiences.\n",
    "\n",
    "# Congratulations! Time to join the Community!\n",
    "\n",
    "Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:\n",
    "\n",
    "## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)\n",
    "This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.\n",
    "\n",
    "## Join the DeepChem Gitter\n",
    "The DeepChem [Gitter](https://gitter.im/deepchem/Lobby) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "04_Introduction_to_Graph_Convolutions.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
